Discriminative Regularization: A New Classifier Learning Method

Regularization involves a large family of the state-of-the-art techniques in classifier learning. However, since traditional regularization methods essentially derive from ill-posed multivariate functional fitting problems which can be viewed as a kind of regression, in classifier design, they usually give more concerns to the smoothness of the classifier, and do not sufficiently use the prior knowledge of given samples. Actually, due to the characteristics of classification, the classifier is not always necessarily smooth anywhere, especially near the discriminant boundaries between classes. Radial Basis Function Networks (RBFNs) and Support vector machines (SVMs), as two most famous ones in the regularization family, have been aware of the importance of the prior information to some extent. They focus on either the intra-class or the inter-class information respectively. In this paper, we present a novel regularization method – Discriminative Regularization (DR), which provides a general way to incorporate the prior knowledge for classification. Through introducing the prior information into the regularization term, DR aims to minimize the empirical loss between the desired and actual outputs, as well as maximize the inter-class separability and minimize the intra-class compactness in the output space simultaneously. Furthermore, by embedding equality constraints in the formulation, the solution of DR can follow from solving a set of linear equations. The * Corresponding author: Tel: +86-25-84896481 Ext. 12106; Fax: +86-25-84498069; E-mail: [email protected] (S. Chen), [email protected] (H. Xue) and [email protected] (Q. Yang) classification experiments show the superiority of our proposed DR.